-
Notifications
You must be signed in to change notification settings - Fork 8
Advanced configuration for Cadence Server #29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Advanced configuration for Cadence Server #29
Conversation
Finished DB Compatibility Tests. This is the pod and commands that I used to test it: apiVersion: v1
kind: Pod
metadata:
name: cadence-debug-pod
namespace: cadence-testing
labels:
app: cadence-debug
spec:
restartPolicy: Always
initContainers:
- name: fix-cert-permissions
image: alpine:3.18
command: ["/bin/sh"]
args:
- -c
- |
# Copy certificates from secret mount to writable location
cp /tmp/certs-source/ca.pem /etc/cadence/ssl/ca.pem
cp /tmp/certs-source/client.pem /etc/cadence/ssl/client.pem
cp /tmp/certs-source/client-key.pem /etc/cadence/ssl/client-key.pem
# Set correct permissions
chmod 644 /etc/cadence/ssl/ca.pem
chmod 644 /etc/cadence/ssl/client.pem
chmod 600 /etc/cadence/ssl/client-key.pem
# Verify permissions
echo "Certificate permissions:"
ls -la /etc/cadence/ssl/
volumeMounts:
- name: mysql-tls-certs
mountPath: /tmp/certs-source/
readOnly: true
- name: mysql-tls-certs-writable
mountPath: /etc/cadence/ssl/
containers:
- name: cadence-server
image: docker.io/ubercadence/server:v1.3.1
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args: ["-c", "while true; do sleep 3600; done"]
env:
- name: KEYSPACE
value: cadence
- name: VISIBILITY_KEYSPACE
value: cadence_visibility
- name: RF
value: "1"
- name: CADENCE_HOME
value: /etc/cadence
volumeMounts:
- name: mysql-tls-certs-writable
mountPath: /etc/cadence/ssl/
readOnly: true
volumes:
- name: mysql-tls-certs
secret:
secretName: mysql-crt
items:
- key: ca.crt
path: ca.pem
- key: tls.crt
path: client.pem
- key: tls.key
path: client-key.pem
- name: mysql-tls-certs-writable
emptyDir: {}
---
# Cassandra Schema Setup
SCHEMA_DIR=$CADENCE_HOME/schema/cassandra/cadence/versioned
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 1h2GcyDEhU --aa "org.apache.cassandra.auth.PasswordAuthenticator" create -k $KEYSPACE --rf $RF
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 1h2GcyDEhU --aa "org.apache.cassandra.auth.PasswordAuthenticator" -k $KEYSPACE setup-schema -v 0.0
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 1h2GcyDEhU --aa "org.apache.cassandra.auth.PasswordAuthenticator" -k $KEYSPACE update-schema -d $SCHEMA_DIR
VISIBILITY_SCHEMA_DIR=$CADENCE_HOME/schema/cassandra/visibility/versioned
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 1h2GcyDEhU --aa "org.apache.cassandra.auth.PasswordAuthenticator" create -k $VISIBILITY_KEYSPACE --rf $RF
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 1h2GcyDEhU --aa "org.apache.cassandra.auth.PasswordAuthenticator" -k $VISIBILITY_KEYSPACE setup-schema -v 0.0
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 1h2GcyDEhU --aa "org.apache.cassandra.auth.PasswordAuthenticator" -k $VISIBILITY_KEYSPACE update-schema -d $VISIBILITY_SCHEMA_DIR
---
# Cassandra TLS Schema Setup (not working, so I will use without SSL)
KEYSTORE_PASSWORD=$(cat /etc/cadence/ssl/passwords/keystore-password)
TRUSTSTORE_PASSWORD=$(cat /etc/cadence/ssl/passwords/truststore-password)
cadence-cassandra-tool \
--ep $CASSANDRA_SEEDS \
-u cassandra \
-pw ZDlnq3sj9G \
--aa "org.apache.cassandra.auth.PasswordAuthenticator" \
--tls \
--tls-ca-file /etc/cadence/ssl/ca.pem \
--tls-cert-file /etc/cadence/ssl/client.pem \
--tls-key-file /etc/cadence/ssl/client-key.pem \
create -k $KEYSPACE --rf $RF
---
# PostgreSQL Schema Setup
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw tdEmPobbF9 --plugin postgres create-database --db cadence
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw tdEmPobbF9 --plugin postgres create-database --db cadence_visibility
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw tdEmPobbF9 --plugin postgres --db cadence setup-schema -v 0.0
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw tdEmPobbF9 --plugin postgres --db cadence update-schema -d ./schema/postgres/cadence/versioned
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw tdEmPobbF9 --plugin postgres --db cadence_visibility setup-schema -v 0.0
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw tdEmPobbF9 --plugin postgres --db cadence_visibility update-schema -d ./schema/postgres/visibility/versioned
---
# PostgreSQL TLS Schema Setup
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw qdeDxXG5Fn --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin postgres create-database --db cadence
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw qdeDxXG5Fn --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin postgres create-database --db cadence_visibility
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw qdeDxXG5Fn --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin postgres --db cadence setup-schema -v 0.0
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw qdeDxXG5Fn --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin postgres --db cadence update-schema -d ./schema/postgres/cadence/versioned
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw qdeDxXG5Fn --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin postgres --db cadence_visibility setup-schema -v 0.0
cadence-sql-tool --ep postgresql.cadence-testing.svc.cluster.local -p 5432 -u postgres -pw qdeDxXG5Fn --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin postgres --db cadence_visibility update-schema -d ./schema/postgres/visibility/versioned
---
# MySQL Schema Setup
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw voxAAfyTLH --plugin mysql create-database --db cadence
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw voxAAfyTLH --plugin mysql create-database --db cadence_visibility
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw voxAAfyTLH --plugin mysql --db cadence setup-schema -v 0.0
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw voxAAfyTLH --plugin mysql --db cadence update-schema -d ./schema/mysql/v8/cadence/versioned
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw voxAAfyTLH --plugin mysql --db cadence_visibility setup-schema -v 0.0
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw voxAAfyTLH --plugin mysql --db cadence_visibility update-schema -d ./schema/mysql/v8/visibility/versioned
---
# MySQL TLS Schema Setup
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw aYtpr7o8as --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin mysql create-database --db cadence
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw aYtpr7o8as --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin mysql create-database --db cadence_visibility
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw aYtpr7o8as --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin mysql --db cadence setup-schema -v 0.0
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw aYtpr7o8as --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin mysql --db cadence update-schema -d ./schema/mysql/v8/cadence/versioned
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw aYtpr7o8as --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin mysql --db cadence_visibility setup-schema -v 0.0
cadence-sql-tool --ep mysql.cadence-testing.svc.cluster.local -p 3306 -u root -pw aYtpr7o8as --tls --tls-ca-file /etc/cadence/ssl/ca.pem --tls-cert-file /etc/cadence/ssl/client.pem --tls-key-file /etc/cadence/ssl/client-key.pem --plugin mysql --db cadence_visibility update-schema -d ./schema/mysql/v8/visibility/versioned ElasticSearch Schema ManualapiVersion: v1
kind: Pod
metadata:
name: cadence-debug-pod
namespace: cadence-testing
labels:
app: cadence-debug
spec:
restartPolicy: Always
initContainers:
- name: fix-cert-permissions
image: alpine:3.18
command: ["/bin/sh"]
args:
- -c
- |
# Copy certificates from secret mount to writable location
cp /tmp/certs-source/ca.pem /etc/cadence/ssl/ca.pem
cp /tmp/certs-source/client.pem /etc/cadence/ssl/client.pem
cp /tmp/certs-source/client-key.pem /etc/cadence/ssl/client-key.pem
# Set correct permissions
chmod 644 /etc/cadence/ssl/ca.pem
chmod 644 /etc/cadence/ssl/client.pem
chmod 600 /etc/cadence/ssl/client-key.pem
# Verify permissions
echo "Certificate permissions:"
ls -la /etc/cadence/ssl/
volumeMounts:
- name: elasticsearch-tls-certs
mountPath: /tmp/certs-source/
readOnly: true
- name: elasticsearch-tls-certs-writable
mountPath: /etc/cadence/ssl/
containers:
- name: cadence-server
image: docker.io/ubercadence/server:v1.3.1
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args: ["-c", "while true; do sleep 3600; done"]
env:
- name: KEYSPACE
value: cadence
- name: VISIBILITY_KEYSPACE
value: cadence_visibility
- name: RF
value: "1"
- name: CADENCE_HOME
value: /etc/cadence
volumeMounts:
- name: elasticsearch-tls-certs-writable
mountPath: /etc/cadence/ssl/
readOnly: true
volumes:
- name: elasticsearch-tls-certs
secret:
secretName: elasticsearch-master-crt
items:
- key: ca.crt
path: ca.pem
- key: tls.crt
path: client.pem
- key: tls.key
path: client-key.pem
- name: elasticsearch-tls-certs-writable
emptyDir: {}
---
# Cassandra Schema Setup
SCHEMA_DIR=$CADENCE_HOME/schema/cassandra/cadence/versioned
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 5Zd7mr2L0H --aa "org.apache.cassandra.auth.PasswordAuthenticator" create -k $KEYSPACE --rf $RF
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 5Zd7mr2L0H --aa "org.apache.cassandra.auth.PasswordAuthenticator" -k $KEYSPACE setup-schema -v 0.0
cadence-cassandra-tool --ep cassandra.cadence-testing.svc.cluster.local -u cassandra -pw 5Zd7mr2L0H --aa "org.apache.cassandra.auth.PasswordAuthenticator" -k $KEYSPACE update-schema -d $SCHEMA_DIR
---
# ElasticSearch Schema Setup
SCHEMA_FILE=$CADENCE_HOME/schema/elasticsearch/v7/visibility/index_template.json
server=`echo elasticsearch.cadence-testing.svc.cluster.local:9200 | awk -F ',' '{print $1}'`
URL="http://$server/_template/cadence-visibility-template"
curl -X PUT $URL -H 'Content-Type: application/json' --data-binary "@$SCHEMA_FILE"
URL="http://$server/cadence-visibility"
curl -X PUT $URL
---
# ElasticSearch TLS Schema Setup
SCHEMA_FILE=$CADENCE_HOME/schema/elasticsearch/v7/visibility/index_template.json
server=`echo elasticsearch.cadence-testing.svc.cluster.local:9200 | awk -F ',' '{print $1}'`
URL="https://$server/_template/cadence-visibility-template"
curl -u elastic:testingPassword --cacert /etc/cadence/ssl/ca.pem --cert /etc/cadence/ssl/client.pem --key /etc/cadence/ssl/client-key.pem -X PUT $URL -H 'Content-Type: application/json' --data-binary "@$SCHEMA_FILE"
URL="https://$server/cadence-visibility"
curl -u elastic:testingPassword --cacert /etc/cadence/ssl/ca.pem --cert /etc/cadence/ssl/client.pem --key /etc/cadence/ssl/client-key.pem -X PUT $URL |
@taylanisikdemir I just pushed this for review. But still pending 2 points:
Let me know if you need something :) |
Thanks. ES issue might need some config changes. Team is looking. |
Server 1.3.2 && Web 4.0.6
Fixed metrics error, the msg is still in the log but at least are working... |
Ready to merge @taylanisikdemir |
…EV/cadence-charts into advanced-configuration
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good at first glance but since it's a huge change I'd like to test it before merging. I will get back to this next week.
failoverVersionIncrement: {{ .Values.config.cluster.failoverVersionIncrement | default 10 }} | ||
primaryClusterName: {{ .Values.config.cluster.primaryClusterName | default "cluster0" }} | ||
currentClusterName: {{ .Values.config.cluster.currentClusterName | default "cluster0" }} | ||
clusterGroup: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default it makes sense to populate clusterGroup
with just current cluster info. This would be out of the box single cluster deployment.
However, when clusterGroup
is populated in values.yml (not unset) then we should just use that here instead of adding current cluster by default. The config would provide complete list of clusters, not just others.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I followed the steps in CONTRIBUTING.md and the services didn't startup. Stuck on init.
k logs cadence-release-frontend-5584f8b696-vfq6k -n cadencetest -c wait-for-schema
...
Waiting for Cassandra schema to be ready...
Traceback (most recent call last):
File "/opt/cassandra/bin/cqlsh.py", line 2354, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/opt/cassandra/bin/cqlsh.py", line 2300, in main
shell = Shell(hostname,
File "/opt/cassandra/bin/cqlsh.py", line 468, in __init__
load_balancing_policy=WhiteListRoundRobinPolicy([self.hostname]),
File "/opt/cassandra/bin/../lib/cassandra-driver-internal-only-3.25.0.zip/cassandra-driver-3.25.0/cassandra/policies.py", line 425, in __init__
File "/opt/cassandra/bin/../lib/cassandra-driver-internal-only-3.25.0.zip/cassandra-driver-3.25.0/cassandra/policies.py", line 426, in <listcomp>
File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
Waiting for Cassandra schema to be ready...
cassandra: | ||
image: | ||
repository: "cassandra" | ||
tag: "4.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have cassandra:4.1.1 by default but schema check is using cassandra:4.0. Is that intentional?
Add comprehensive configuration support for Cadence server
Summary
This PR introduces a complete configuration system for the Cadence Helm chart, enabling users to customize all aspects of the Cadence server deployment through
values.yaml
.Changes
Added comprehensive
config
section invalues.yaml
with support for:Enhanced global configuration:
global.env
andglobal.secretEnv
examples for the use of S3 or GCPglobal.tls
for centralized TLS certificate managementConfigMap integration: Configuration is automatically converted to YAML and mounted in server deployments
Benefits
Backward Compatibility
All changes are backward compatible. Existing deployments will continue to work with default configurations.
Test Checklist
Default Deployment - Basic deploy with defaults



Secrets handled - Database Secrets handled correctly in configMap and application


Log Use Env - Check if the boolean allows to use different log level for each component.




Database Connectivity
MySQL basic connection - Basic connection with credentials



Deployed Bitnami MySQL version 10.3.0 with appversion 8.0.37.
MySQL with TLS - TLS encryption enabled






Testing with latest version of mysql for auto-generation of tls, chart version: 13.0.2 and appversion 9.3.0
Enabled tls generation in mysql helm chart
Using a debug pod that I will share in a comment I created the schema with TLS.
I changed this cadence values for test it
Working fine:
PostgreSQL basic connection - Basic connection with credentials



With schema job disabled, I started a debug pod that I will share in the PR for apply the schema manually. I just changed this values:
First I deployed Bitnami Postgresql Helm chart 16.7.0 with app version: 17.4.0 and then this values:
PostgreSQL with TLS - TLS encryption enabled







Deployed postgresql with tls auto-generated
Started schema with debug-pod that I will provide in a comment when it's finished all the test cases.
Changed this values for testing:
And working correctly:
IMPORTANT: I will let the mode example by default in TLS because PostgreSQL requires the permission to be restrictive.
Cassandra basic connection - Basic connection with credentials


With schema job disabled, I started a debug pod that I will share in the PR for apply the schema manually. I just changed this values:
First I deployed Bitnami Cassandra Helm chart 10.4.0 with app version: 4.1.1 and then this values:
Cassandra with TLS - TLS encryption enabled






Enable that in cassandra values for generate secrets with certs
Now the cadence values changed for SSL with Cassandra
Setup job and cassandra included deployment disabled.
Schema Setup was run manually without TLS, schema setup is for next PR so I wont focus on that right now.
Working all correctly
IMPORTANT: serverName value will be default to cassandra because if we let empty, we will have this error connection:
2025/06/23 10:51:43 error: failed to connect to 10.100.185.52:9042 due to error: tls: failed to verify certificate: x509: cannot validate certificate for 10.100.185.52 because it doesn't contain any IP SANs
Advanced Features
Elasticsearch + Kafka integration - Advanced visibility with Elasticsearch




Deployed Bitnami ElasticSearch helm chart 21.6.3 with appversion 8.18.0.
Deployed strimzi kafka version 3.9.0
Deployed Cassandra from previus tests.
All runing correctly:
Values.yaml for testing cadence with elastic:
Schema has been loaded manually, I will provide the commands in a commentary here but the next PR with schema jobs will include that :)
Kafka publish is working:
But I cannot load from Cadence-Web and it's giving an Internal Server Error 400, I tried to use a version of ElastciSearch 7.X and still the same issue. I will try to see what's wrong....
Elasticsearch with TLS integration + Kafka with TLS + SASL - Advanced visibility with Elasticsearch TLS
Blocked temporaly
S3 archival - History archival to S3





I didn't need the env variables for credentials, but if the Bucket have permission restrictions it will be needed.
This values:
GCS archival - Visibility archival to Google Cloud Storage
I don't have GCP available for the testing. But the schema follows this README:
https://github.com/cadence-workflow/cadence/blob/v1.3.0/common/archiver/gcloud/README.md
So it will work correctly.
Filestore archival - Local file archival



Tested with this values:
Monitoring & Metrics
{"level":"warn","ts":"2025-06-19T12:01:36.325Z","msg":"error in prometheus reporter","service":"cadence-frontend","error":"a previously registered descriptor with the same fully-qualified name as Desc{fqName: "cadence_latency", help: "cadence_latency histogram", constLabels: {}, variableLabels: [cadence_service operation domain caller transport]} has different label names or a different help string","logging-call-at":"metrics.go:151"}
It's an issue in cadence 1.3.1. Should be fixed in 1.3.2 http://github.com/cadence-workflow/cadence/pull/6963
Still getting the error msg but metrics are working.
I don't have a statsD installation so I just check enabling and the schema is reading it, so it's okey.
Advanced Cluster
Validated in Cassandra with TLS check.
Validated in Secrets handled check and in rest of checks.
Validated changing the default visibility to db from es (elasticSearch) adding the dynamicConfig writeVisibilityStoreName readVisibilityStoreName. You can try yourself to remove it, and you will see that it's used the default "ES" value and cadence won't work with defaults.